Brownian Bandits

We consider a multi-armed bandit whose arms are driven by Brownian motions: the state of arm i is modeled as a one-dimensional Brownian motion B, i = 1, . . . , d. At each instant in time, a gambler must decide to pull some subset of these d arms, holding the others fixed at their current state. If arm i is pulled when B is in state xi, the gambler accumulates reward at rate ri(xi). The goal is to find a strategy that maximizes the accumulated discounted reward over an infinite horizon (with discount rate λ). Let Γi(xi) = sup τ>0 Exi ∫ τ 0 eri(B i t)dt Exi ∫ τ 0 e−λtdt , where the supremum is over all stopping times τ of B. Put Mi = {x ∈ IR : Γi(xi) = max j Γj(xj)}. From prior work, one expects that an optimal control is to pull arm i when the state of the bandit belongs to Mi. Equivalently, the optimal strategy follows the leader among the processes Γi(B ), i = 1, . . . , d. Such results have been established for bounded monotone reward functions. In this paper we extend the scope to cover certain unimodal functions. At the same time, we develop a framework within which general rewards and diffusions can be studied.